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Introduction 

This paper is the culmination of a series dedicated to the problem of constructing 
explicit analytic models for small unitary representations of certain semisimple 



Lie groups. (See §1), §2|, |S3[, |DSl|], ^Sg] and also |S^], 



The groups G that we consider are those that arise from real semisimple 
Jordan algebras via the Tits-Koecher-Kantor construction. Such a G is char- 
acterized by the fact that it admits a parabolic subgroup P — LN which is 
conjugate to its opposite P = LN, and for which the nilradicals N and N are 
abelian. We identify N and N with their Lie algebras n and n via the expo- 
nential map, and moreover we identify n with n* via a suitably rescaled Killing 
form. The space n is naturally endowed with a real semisimple Jordan algebra 
structure, which is isotopic to the one giving rise to G and P. We write n for 
the rank of n as a Jordan algebra. 

In this situation, the Levi component L has a finite number of orbits on 
n; each orbit has a rank < n, and carries a measure dfi which transforms 
by a character under L. For the non-open orbits (rank < n) this measure is 
unique up to scalar multiple; for the open orbits (rank = n) there is a one- 
parameter family of such measures. For each non-open orbit O we consider the 
Hilbert space Ho — C^{0,dii). By Mackey theory this space carries a natural 
irreducible unitary representation no of P, and we consider the following two 
problems: 

• Extend tto to a unitary representation of G. 

• Decompose ttqi ® • • • (g) ttos, for rankO^ + ... + rankO^ < n. 



For Euclidean Jordan algebras the first problem was solved in pll and |S2|, 



and the second problem was solved in [DS1|. Therefore in this paper we restrict 
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our attention to non-Euclidean Jordan algebras. The case of rank 1 orbits in 



non- Euclidean Jordan algebras was completely settled in |DS2|. It turned out 
that for rank 1 orbits in certain rank 2 Jordan algebras the representation ttq 
cannot be extended to G. We shall call these orbits inadmissible] they arise for 
the groups G = 0{p,q), p ^ q and are connected with the Howe-Vogan "no-go" 
result for minimal representations of such groups (0]). For rank 1 orbits in the 
remaining Jordan algebras we establi shed the desired extension and calculated 



the tensor product decomposition in |DS2] 



In this paper we consider the general case, and prove the following results: 
First suppose O is an admissible non-open orbit in a non-Euclidean Jordan 
algebra N. 

Theorem 0.1 For O as above, the representation ttq of P extends to an irre- 
ducible spherical unitary representation of G on "Ho ■ 

Now suppose O^,... ,0* are admissible non-open orbits in N such that 



rankO^ + ...+rankC < n. In subsection 4.2 we define a reductive homogeneous 
space G' /H' , essentially the generic fiber of the addition map from x . . .xO^ 
to N , and consider the decomposition of the quasi-regular representation 



C^{G'/H')^ I m{a)adp{a) 
Jg' 



where m(cr) is the multiplicity function and dp{a) is the Plancherel measure. 

Theorem 0.2 Let , . . . ,0* and G',H' be as above; then there is a map 
from the H' -spherical dual of G' to the unitary dual of G such that 



ttqi (g) • • • (g) TTos = / m{a)9{(j) dp{cr). 

JG' 

Our approach to these results requires consideration of three different kinds 
of groups, each with its own flavor of representation theory. These are: 

• Harish-Chandra modules for semisimple Lie groups. 

• Operator algebras for parabolic subgroups. 

• Fourier analysis for abelian nilradicals. 

The Harish-Chandra theory was studied in The necessary operator- 

algebraic results (C*-algebras, von Neumann algebras) were already obtained 
in |DSI| and | DS2| ]. Thus the missing ingredient, which is provided by this 



paper, involves abelian Fourier analysis. The key result. Proposition shows 
that a certain function (eventually, the "spherical" vector in ttq) belongs to 
£^{0,dfj,). For rank 1 orbits this result was obtained in [DS2| by establishing 



a close connection between this function and a certain one-variable Bessel K- 
function. The required £^-estimate was then deduced from a precise knowledge 
of the singularity of the Bessel if-function at 0. 
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For higher rank orbits, we expect that there should exist a similar connection 
between the spherical vectors and multivariate Bessel i^T- functions. However in 
order to exploit this connection one would have to first develop the theory of 
such functions, possibly along the lines of the theory of the multivariate Bessel 
J- functions of |0p|. While we feel that the connection with multivariate Bessel 
-fC-functions is of interest and should be pursued further, in the present paper 
we follow a different approach that allows us to obtain the desired estimate 
directly, obviating the nee d to first study Bessel functions. The key here is a 
"stability" result (Lemma 3.12 ) which transfers the problem from a non-open 
orbit to a related problem on the open orbit for a smaller group. The open orbit 
problem turns out to be easier to solve. 

This approach was inspired in part by a recent paper of Shimura [3h|. We 
wish to thank L. Barchini for drawing our attention to this paper. 



1 Preliminaries 



In this section we recall basic facts about the Tits-Kantor-Koecher construction. 
All results of this section are well-known. More details may be found in [KS|, 
[DS2| and in the references therein (in particular, |BK] and [Lo|). 



1.1 The pair {G,P) 

The Tits-Kantor-Koecher construction associates to a real simple Jordan alge- 
bra, a pair (G, P) , where G is a real simple Lie group with Cartan involution 
9, and maximal compact subgroup K; and P is a parabolic subgroup with 
Levi decomposition LN, say. In the context of Lie theory, these pairs can be 
characterized as follows: 



1. is abclian. 

2. P is G-conjugate to its opposite parabolic P = d{P) ~ LN. 

Conversely, in the above situation one can endow N with the structure of a 
real simple Jordan algebra, unique up to the choice of an identity element. 

In view of the classification of real simple Lie groups and their parabolic 
subgroups in terms of the possible restricted root systems, it is an easy matter 
to determine the above pairs. The two conditions correspond to simple restricted 
roots a such that 



1. a has coefficient 1 in the highest root 

2. a satisfies a — —WQa for the longest element wq of the Weyl group. 

The conditions 1. and 2. each give rise to a symmetric space denoted by K/M 
and L/H respectively, and much of the relevant information about the Jordan 
algebra and the associated pair (G, P) can be described in a simple and coherent 
manner in terms of these symmetric spaces. This makes it possible to have a 
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uniform discussion for the most part, with only some occasional arguments 
requiring case-by-case considerations. 

In the next few sections we describe these spaces and conclude by giving a 
complete list of examples. 

We follow the practice of denoting the real Lie algebras of various Lie groups 
by the corresponding fraktur letters; with the exception of p which will denote 
instead the —1 eigenspace of 9 in the Cartan decomposition g = t © p. 

1.2 The symmetric space K/M 

Condition 1. implies (and is equivalent to the assertion) that the subgroup L is 
a symmetric subgroup of G, and M ~ K n L is a symmetric subgroup of K. 
Let t be a maximal toral subalgebra for the compact symmetric space K/M , 
i.e. a Cartan subspace for in the orthogonal complement of m in 6. The real 
rank of as a Jordan algebra is n = dimg t. 

The roots of tc in gc always form a root system of type C„, and we fix a 
basis {71, 72, • ■ ■ , 7ti} of t* such that 

I](tc,gc) = {±(7.±7,)/2,±7,}- 
For the subsystem S = S(tc,tc)) there are three possibilities: 

A„_i = {±(7, - 7,)/2}, D„ = {±(7» ± 7,)/2}, and C„. 

The first of these cases arises precisely when is a Euclidean Jordan algebra. 
This case was studied in , therefore we restrict our attention to the last two 
cases. If E is C„, there are two multiplicities, corresponding to the short and 
long roots, which we denote by d and e, respectively. If E is Dn, and n ^ 2, 
then there is a single multiplicity, which we denote by d, so that Dn may be 
regarded as a special case of C„, with e = 0. 

The root system D2 » Ai x Ai is reducible and there are two (possibly dif- 
ferent) root multiplicities. In what follows, we explicitly exclude the case when 
these multiplicities are different. This means that we exclude from consideration 
the groups 

G = 0{p,q),N = W-'^''-\p^q). 

When the two multiplicities coincide {p = q), we once again denote the common 
multiplicity by d. 

1.3 S'-triples and the Cayley transform 

The discussion of the various cases can be made uniform by emphasizing the 
special role played by a family of n commuting SL2^s or S'-triples, together with 
the associated Cayley transform. 
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For the Lie algebra s[2(C), we define 
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H = i 

The Cay ley transform is the automorphism of sl2(C) given by 



TTi ni 1 

c = exp ad — [X + Y) = exp ad — [x + y) — Ad — ^ 



1 i 
i 1 



It satisfies 



c{X) = x,c{Y)=y,c{H) = h. 



Now, as remarked earlier, the root system X^(tc, flc) is of type C„. Moreover, 
the various compact and non-compact root multiplicities are as follows: 

dime±(^,±^^.)/2 = d, dimp±(^,±^^.)/2 = d 
dimt±^. = e, dimp±^. = 1 

We fix holomorphic homomorphisms 

<i>j :s[2(C) — > gc, j = l,...,n 

such that each '^j{X) spans p^., and wc write 

= = "ifjix), Vj = *,(j/),etc. 

The images of commute with each other and we also write 

X = ^ Xj, X = ^ a;^, y = ^ y^, etc. 
j j j 

The Cayley transform of g is the product 

c = exp ad ^ (x + y) = exp ad ^ (X + Y). 

We write a = c {it) for the Cayley transform of it. This is the abelian subalgebra 
of g spanned by /ii, • • • , 

1.4 The symmetric space L/H 

Let H C L he the stabilizer of y € n, then condition 2. implies (and is in fact 
equivalent to the assertion) that L/H is a symmetric space. The involution a 
for this symmetric space consists of conjugation by a suitable element oi K — 
corresponding to the element wq of condition 2. 
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Example. If G = 02n.2n, then L = GL-znC^) and N is the Jordan algebra 
of 2n X 2n real skew-symmetric matrices, and H = S'p„(R). 

In the present situation L/H is always non-Riemannian; and moreover a is 
the corresponding Cartan subalgebra in the usual sense. In other words, if we 
consider the Cartan decompositions for 9 and a 

l = m + r, [ = () + q; 

then a is a Cartan subspace in q n r. 

Since a = c (it), the roots of a in g are 

E(a,0) = {±ei±£j,±2£j} where = o 

Moreover it is easy to see that 
T.{a, [) = {±{e, - Ej)} , S(a,n) = {e^ +ej,2ej}, S(a,n) = {-e, - ej,-2ej} . 
We now remark that the weight 

^ El + 62 + ■ ■ ■ + Sn 

extends to a character of [. The easiest way to see this is to consider for a in a, 
tradn(a)= -2d^ (e^ + e^) - (e + 1) ^ 2£j (a) ^ -2riy (a) . 

Thus we can define 

i^{l) = - (^Y^ tradn (/) for I e I, where r = d(n - 1) + (e + 1). 

Similarly, we define a corresponding positive character of L by 

e" (0 =r = |dct AdnC^''^^''' for I in L. 

Considering an appropriate power of the determinant we obtain a corresponding 
positive character of the groups P and P, which we write as 5 i-^- e" (g), or as 
9 ^ 9"- 

To complete the connection with the Jordan structure, we note that the 
Jordan norm on n is a polynomial function which transforms by the character 
e"^'^ of L. Finally we observe that the Killing form on g gives a pairing between 
n and n which we rescale by setting (xi, yi) = 1. 



2 Integral formulas 



It is an immediate consequence of Corollary 5.3 that the L-orbits in n carry 
equivariant measures. More precisely, we write e'^ for the positive character of 



L defined in 1.4 and let r — d{n — 1) + (e + 1) be as before. Then we have 
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Lemma 2.1 1. The Lehesgue measure dX on n is e^^'^ -equivariant for the 
L-action. 

2. The rank k-orbit carries an e^'^^'^ -equivariant measure djjL = djjLk- 

The (easy) proof of this lemma is postponed to the next subsection. We 
now describe a "polar coordinates" expression for these equivariant measures. 
Let O be the rank k orbit, O = L ■ {yi + . . . + yk) ■ In |Lo| it is shown that the 
elements 

{Zlj/l + . . . + ZkVk \zi> Z2> ■■■> Zk>0) 

give a complete set of orbit representatives for the action oi M ~ L{^K on the 
rank k orbit. Accordingly, we write Ck C M'' for the cone 

Cfc = {Z = (Zl, 22, ■■■ ,Zk) \ zi> Z2> ■■■> Zk>Q}\ 

and for m in Af, z in Cfe we write 

TO • z = AdTO (zij/i + . . . + Zfej/fc) e n. 
For z in Ck we introduce the notation 



l<i<j<k 



dZi 



■ 1 ^3 



where each dzj denotes the Lebesgue measure on R. 

The main results of this section are summarized in the following two propo- 
sitions. 

Proposition 2.2 Let dX be the Lebesgue measure on n, then 



fdX = c 



f{m ■ z)dm 



d^,z where d^,z — [PnT~^^ [Vn]'^ d^ 



Proposition 2.3 Let dfi be the equivariant measure on the rank k orbit O, then 



fdn = c 



f{m ■ z)dm 



dkz where d^z = [Pr'''^'"*^] 4 



The scalars c appearing in the above integral formulas are independent of / 
and depend only on the normalization of the measures dX and d/i. 

For subsequent purposes we need to consider the Lebesgue measure on n as 
well as n. For to in Af, z in C„, we write 



m o z = Ad m (ziXi + . . . + z„a:„) G n . 



(1) 



Since the Cartan involution 9 gives a linear isomorphism between n and n, 
satisfying 9{moz) ~ m ■ z, the following result can be derived immediately 
from Proposition 2.2. 
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Corollary 2.4 Let dX be the Lebesgue measure on n, then for all functions f 
on n 



fdX^c 



c„ La/ 



/(m o z)dm 



dfZ where d*z — [Pn]''^^ iVn]"^ dnZ. 



The proofs of the propositions will occupy the rest of this section. 



2.1 Stabilizers and equivariant measures 



In this subsection, we prove Lemma For this we need to first determine the 
stabilizer of the point yi + . . . + in the rank k orbit. If A: = n, the stabilizer is 
the symmetric subgroup H described previously. We now discuss the remaining 
orbits. To simplify notation we fix k and write 

= yi + ■ • • + yfc- 

In Jordan algebra terms, is a Peirce idempotent and considering the 1 and 
Peirce-eigenspaces of y-"^, we obtain smaller Jordan algebras rii and no with 
identity elements y^ and y'' ~ yu+i + • ■ • + yn, respectively. The corresponding 
structure groups Li and are naturally subgroups of L. Subgroups of Li and 
L{) will be distinguished by subscripts 1 and 0, respectively. For example. 

Ml = M n Li, Mq = M f] Lq, Hi = H f] Li, ai = a n [i. 

Thus Hi is the stabilizer of y^ in Li, and the full stabilizer of y^ in L is given 

by 

S=iHiX Lo) ■ U, (2) 

where U is the abelian subgroup whose Lie algebra u is spanned by the root 
spaces {^<i<k<j<n). Then 

[ = 5+ (qi + u) , where qi = q n Ii, u = 0u. 



Proof, (of Lemma 2.1) From the calculation in the previous section, it 
follows that the Lebesgue measure dX on n is equivariant by the character 

r; (/) = |det Adn /P' = e^™ (l) = for I in L, 

which proves the first part of the lemma. 

For the second part, we consider the stabilizer of + . . . + First suppose 
k — n; in this case the stabilizer H is semisimple, and hence 

|detAd(,/i| = 1 for all he H. 
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Since e'^ln = 1 as well, it follows from Corollary 5.3 that for each real t, the 
open orbit carries a measure which is e'^'-equivariant. The measure dfj, is simply 
the special case t = 2dn. 

Now suppose k < n, then s = f)i + [o+u and it is easy to see that tr [ad^ (z)] = 
for z in [)i and u. Thus we can assume that z is in [q, and then tr [ad[,i+[(, (z)] = 
0, hence tr [ad., {z)] = tr [ad^ (z)]. To calculate this, it suffices to consider z in 
ao, and then we get 

tr [ads (z)] = tr [adu (z)] = ^ 2d (— + Sj) (z) 

l<i<fe<j<ri 

= -2d {n-k)[ei + ...+ Bk] (z) + 2dk [sk+i + . . . + e„] (z) . 

Now the weights ei, . . . , efc restrict trivially on oo, hence we see that the required 
trace is the restriction of 2dkv to s. Thus we get 



[dot Ads s| 



„2dkii 



and the result follows by Corollary 5.3 



2.2 Jacobians for homogeneous spaces 

In order to prove integral formulas for homogeneous spaces, we need a method 
for calculate the Jacobian of a diffeomorphism between such spaces. It is conve- 
nient to work with the more flexible notion of a local dijfeomorphisms between 
X and y, by which we simply mean a diffeomorphism between open subsets of 
X and Y. We also introduce the notation 

to represent the situation where X and Y are smooth manifolds; x and y are 
points in X and Y; F is a diffeomorphism from an open neighborhood of x to 
an open neighborhood of Y such that F {x) = y. 

We now consider the following situation: Suppose X and Y are homogenous 
spaces for groups G and H, and dx and dy are regular measures on this spaces 
which are equivariant for characters 7 (g) and rj (h) respectively. We choose two 
points xq G X and yo & Y , and fix linear bases for the tangent spaces at these 
points. 

Lemma 2.5 Suppose that in the above situation we have a local diffeormor- 
phism F : X ^ Y . Then for all x in the domain 0/ F we have 

Jf {x) = cry {h) 7 {g) |det DhFg \ , 

where c is a scalar independent of x; h €z H and g (z G satisfy 

g -xq^x, h- F{x) ^yo 

and DhFg '■ Tx„X — > Ty^Y is regarded as a matrix via the above bases. 
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Proof. We fix x and write y = F (x) . Tlien we hiave local difFeomorphisms 

{X,xo)^{X,x)^iY,y)^{Y,yo). 
By formula ( |l6| ) we get 

JhFg {xo) = Jg {xo) Jf (x) Jh (y) = 7 (5)"^ Jf (x) t] {h)~^ . 
On the other hand by Lemma we have 

JhFg (xo) =c\detDhFg ixQ)\ 
for some positive scalar c, independent of hFg. The lemma follows. ■ 

2.3 Integral formula for the Lebesgue measure 

We now apply the results of the previous subsections to prove the integration 
formulas on the L-orbits. In order to do this we first need to fix bases for various 
subspaces of I which are compatible with the actions of 9 and a. We start with 
the weight decomposition 

where is the centralizer of a. The involutions 9 and a act by —1 on a, thus 
the space l*^ is invariant by 9 and a; considering their eigenvalues we have the 
decomposition 

[0 ^ J++ _^ [+- ^ [-+ ^ [ — 

where 1++ = 1° n (m n t]), = [° n (m n q) etc. Note of course that [ = o. 
On the other hand, the root spaces P are not stable under 9 and cr, in fact each 
involution maps P to However the involution t = a9 = 9a does stabilize 
these spaces. Considering the eigenvalues of r we have a decomposition 

where P'+ = I n (m n f) + r n q), P'" = P n (m n q + r n ()) (cf. 8.1]). 
In the present situation, we always have 

dim 1"'+ = dim C'- = d. 

This fact can be checked eas ily f or each of the examples in the table below, from 
the lists of multiplicities in [pS| . We fix bases 

We now turn to the proof of Proposition |2.2| . The rank n orbit L/H is open 
and dense in n and its complement has measure 0. Since L/H is a symmetric 
space of type the "multiplication" map 

F :rn X a i—>- ma^^ 
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gives an diffeomorphism between [M/M'\ x and L/H, where M' is the 
centrahzer of A in M n H and = exp a"*" with 

a+ = {ci/ii H h c„/i„ I ci > . . . > c„} 

We regard as a local difFeomorphism between the homogeneous spaces [M/M'\ x 
A+ = [M X A]/ [M' X {!}] and L/H. By Corollary U[ the first space carries 
an invariant measure dm x da, while the second sp ace carries the Lebesgue 
measure dX which is e^'''^-equivariant by Lemma 2.1. The main result is the 
following Jacobian computation. 

Lemma 2.6 In the above situation there is a scalar c such that 

i<j 

Proof. We fix the natural base points xq = 1 x 1 £ [M/M'] x A and 
— 1 £ L/H and apply Lemma 2.5 from the previous subsection with 

g = m X a, h — am~^ 

Then we have 

hFg im' , a') = hF [mra' , aa'^ = h (jnm' {aa') = (am'a^^) a'~^ 

Therefore the differential Dhpg : m/m' + a ^ [/f] is given by 

Ta {v, w) := Ad {a)v + w (mod f)) (3) 

This shows that 

JF,drnxda,dx(jn X a) ~ fl^™ |det Ta\ 

for some fixed choice of bases for m/tn' + a and l/t). Since r = d{n — 1) + e + 1, 
the result follows from the next lemma. ■ 

Lemma 2.7 For the map Ta defined in formula (Qj we have 
|det Tal ~o^2'^("-i)'^ Jll 



Proof. In order to compute the determinant, we choose convenient bases 
for [/() and m/m' + a. We have w q, and so we may choose the basis 

} , { - ext^] , [xr + exr] I a > 0} ; 

For the space m/m' + a, we note that m' = 1++ in our earlier notation, and 
thus a basis is given by: 

{x^'- } , { + , [xr + oxr] I « > 0} 
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We now claim that 



T,.-X 



X 



ex 



a: + l 



. [Xr + OX^^-] = ^^±^ [Xr + 0Xr] (mod t)) 
The first equaUty is obvious. For the second, we calculate 



Ta ■ K-+ + eX^'+] = Ad (a) . [X^-+ + eX^-+] = a"X^'+ + a'^OX^ 



k 



-,a ^ — a 



= - (mod 0) . 

The third equality follows by a similar calculation. 

Since there are c? vectors in each of the sets {X'f^''^ + 9X^''^^ , {X^" 
we get 



■ ex^'+] 



exD 



ldetT„ 



n 

i<j 

ni 

i<j 



Ofi Sj _ ^Sj Si 



n| 



-2(ei+e,) (Ae 



n 

2<j 



,4ei „4e,- 



We can now prove Proposition 



2.2 



Proof, (of Proposition p.2[ ) Since the map is a diffeomorphism between 
[M/M'] X and i/i?, by the Jacobian calculation, we get 



[ fd\= [ fdX = c [ [ f (ma-A dfn 

Jn Jl/H Ja- Jm/M' ^ ' 



I M/M' 



For a suitable normalization of the Haar measures, the innermost integral can 
be rewritten as J^^ / {ma^^^ dm. Also for a = exp (ci/ii + • • • + c„/i„) in A"*", 
we have 



a-i = a • y =e^''^yi + • • • + e^'^-j/n. 



Thus if we identify and C„ via the map 

exp(ci/ii + hc„/i„) < — > (e^'^S- ■ • ,6^^=") 
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then we get 

ma~^ <^ m ■ z, a^" ^ P„, \a'^^' — a*^^' | <-> V^j, da ^ d^: 



Making these substitutions in the integral formula, we obtain the desired result. 



2.4 Integral formula for the equivariant measure 

We now turn to the proof of Proposition For the case of the measure d^i on 
the open orbit, the calculations are exactly the same as in the previous section, 
the only differ enc e being that the measure is e^''"''-equivariant. Thus arguing 
as in Lemmas 2.6 and 2.7 we obtain that 



Now arguing as in the proof of Proposition 2.2 we obtain Proposition 2.3 for 
k = n. 

Now consider the case k < n. We r ecall the notation — yi + . . . + 
yk,Mi,Mo, etc. introduced in section 2A_ Now writing M{ = ^Minifi(fli) and 

= ai n 0+ etc., we get a local diffeomorphism : M/{M[ x Mq) x 
= L/S given by 



F^{m, a) — ma ^ 



Arguing as in Lemma 2.6, we obtain 



where : m/ [m'l + mo) + ai ^ [/s is given by 

TI {v, w) = Ada (w) + w (mods ) 

and we compute the determinant for some choice of basis for the two sides. We 
choose these bases in the following manner. We have ^i-module isomorphisms 



m 



mi 
m' 



m^ + mo mi + mo 

[/s w u + qi. 

Thus the map decomposes as a direct sum = To + Ti with 



To 



m 



mi + mo 



u, Ti : ^ + 01 



qi- 



The calculation for jdetTil is the same as in Lemma 2.7, applied to a smaller 
Jordan algebra of rank k. This gives 



|detTi|= \[ |a2e,-2e, _^2s,-2s.|d^^-2d(fe-l)(ei + ...+sO 

i<j<k i<j<k 



4e, 



-,4e, 
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To calculate detTp we use the following bases for m/ (mi + mo) and u 
mi + mo 

u : {X'^'^ \ a — Ei ~ Sj, i < k,j > k] 
The action of Tq is given by 

ad a • + eX^^"^] = a"Xf + a-"6lXf = a"Xf (mod s) 

Since for a in Ai we have a'^^ = 1 for j > fc, we get 

,e,-ej |2d _ 2d(n-fe)(ei+---+efc) 



|detro|= n 

i<~k.j>k 



a 



Thus we get 

^ ^2<i(n-fe+l)(£i + .--+efc) Jl |^4£i _ ^4£, 
'i<j<fe 



Now arguing as in the proof of Proposition 2.2 we obtain Proposition 2.3 for 
k < n. 



3 Estimates for spherical vectors 

We can relate the P-representation ttq to a unitarizable submodule of a certain 
degenerate principal series for G, which is described as follows: If x is a character 
of L, we write (tt^, I{x)) for the degenerate principal series representation Ind^x 
(unnormalized smooth induction); thus 

Hx) = {/ e C°°(G) !/(/%) - x(0/(.9) forleL,neN,geG} 

and the group G acts by right translations. By virtue of the Gelfand-Naimark 
decomposition G « PN , functions from /(x) are determined by their restriction 
to N. Combining this with the exponential map we can identify /(x) with a 
subspace E{x) of C°°(n) . We refer to this as the noncompact picture. 

For t e M, we write I{t), E{t) for lie*"), Eie*"); more generally, if e : L ^ T 
is a unitary character, we write Ijt, e), E{t, e) for I{e*'^ (g) e), E{e^'^ ^ e). These 
principal series were studied in [p3| via the "Cayley operator" D which is the 
constant coefficient differential operator on n, whose symbol is the Jordan norm 
polynomial (f>. Powers of D are intertwining operators for the principal series, 
and their eigenvalues on the various i^T-isotypic components are given by the 



Capelli identity of |KS| 



E{t) is a spherical representation of G and we write $t for the if-spherical 



vector. Among the results obtained in |S3| is that for 
Xfc-e-'"^ ,n-l; 
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the space E{—dk) contains a unitarizable spherical submodule. We need to 
study the Fourier transforms of the corresponding spherical vectors 



fc = 1, . . . ,n - 1. 

For this we identify n with the dual of n* via the normalized Killing form from 
subsection Also we fix A: < n, write $ for the spherical vector <^^dk, and 
write (O, d/i) for the rank k orbit in n together with its equivariant measure 



described in Lemma 2.1. The main results are 



Proposition 3.1 The measure $dA is a tempered distribution on n and there 
exists an Ad -invariant function g in C^{0,d^) such that 

<^d\ = gd^. 

Proposition 3.2 For k < n, one has g G C^{0, dfi). 

We prove these propositions in the next few subsections. The strategy is 
as follows: Let us write ^k,n for the function ^-dk, in order to emphasize 
dependence on n as well as k. Now although the above results are false in 
general for the open orbit (k — n), nevertheless we can prove the desired results 
by reducing to a slightly weaker estimate for k = n, which turns out to be true, 
and somewhat easier to prove. We establish this result in the next subsection 
and then outline the reduction procedure in the two following subsections. We 



[DS2| 



then deduce Theorem 0.1 from Proposition 3.2 by arguments similar to |S1| and 



3.1 Estimates for the open orbit 

As indicated above, we first consider the function 

We need appropriate £^-estimates with respect to the Lebesgue measure dX 
on n for the function \1/ and its derivatives. The "straightforward" estimate is 
actually false for the group Spn{C), but it does work for the other groups G in 



the table in subsection 5.2 . Thus we formulate two results, one for G ^ Spn(C) 



and the other for all groups: 

Proposition 3.3 For all groups G other than Spn{C), we have $ e C^{n, dX). 



Proposition 3.4 For all groups G and for allm > 1, we have D™^ E C^{n, dX). 



For each t, the function $t is Af -invariant, and is therefore determined by 
the restriction to the subspace {zixi + • • • + ZnXn} C n; we start by giving an 
explicit formula for the restriction. 
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Lemma 3.5 We have 

n 

$t(TO o z) ^ ]^(1 + ) 2 for all m in M. 

i=l 



Proof. For the group G = SL2 (K) this is a straightforward calculation 
which we leave to the reader. In the general case, we view $ as a function on G 
which is right iC-invariant, and left P-equivariant with character e*''. We now 
restrict $ to the subgroup SL2 x • • • x SL2 corresponding to the S'-triples of 
section L3. This restriction is right 5*02 x • • • x 502-invariant, and left Bx ■ ■ - x 



i3-equivariant with character e'^'^ = e'*^^ x • • • x e^^". Thus applying the SL2- 
calculation to each factor, we conclude that the restriction to zixi + • • • + ZnXn 
is given as in the statement of the lemma . ■ 

Combining this with Corollary |2.4| we obtain the following estimate 

Lemma 3.6 For t < - [d (n - 1) + (e + 1) /2], we have $f G £^(n, dX). 



Proof. Combining the previous lemma with Corollary 2.4, we get 
J \^t\^ dX = J 'i{j{z)dzidz2 ■ ■ ■ dzn, 

where 

n 

Hz)^llzni+zfy n i^'-^j)'- 

j=l l<i<j<ji 

Expanding {zf — ^f)'*! we can write V'(^) as a combination of terms of the form 

n 

Y[ (1 + 2?)* where each k, < 2d (n - 1) . 

i=l 

Each of these integrals is a product of one-variable integrals which converge if 

'x^+2'^("-i)(l + a;2)*dx<oo. 
This happens if 2t + e + 2d {n — 1) < —1, which proves the lemma. ■ 

Corollary 3.7 If f € E{t,e) for some t < - [d{n - 1) + (e + 1) /2] and T> is 

any constant coefficient differential operator, then we have Df G C^{n,dX). 

Proof. The group G acts on I{t, e) by right translations. The corresponding 
action of the Lie algebra q in the noncompact picture E{t, e) is by polynomial 
coefficient vector fields on n. The action of x G n is independent of {t, e) and 
is given simply by the directional derivative in the direction x. In particular. 
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the space E(t,s) is invariant for the action of constant coefficient differential 
operators. 

Thus /' = Vf also belongs to E(t,e). Thus /' is the restriction to N of 
a P-equi variant smooth function on G. Since G = PK, any such function is 
determined by its restriction to K . The constant function 1 on K corresponds 
to the spherical vector $4 in I{t). Thus if c is the maximum of |/'| on then 
we have |/'| < c^t^ and the corolla ry fo llow s fro m the previous lemma. ■ 

We can now prove Propositions 3.3 and 3.4 (for G ^ Spn (C). 

Pro of, (of Propositions 3.3 and 3.4 for G 7^ Spn (C)) From the table in 
Section 5^ we see that in every case except G ~ Spn{C), we have 2d > e + 1. 
Consequently we get 

-dn < - [d (n - 1) + (e + 1) /2] . 



Proposition 3_^ now follo ws f rom Lemma |3.6| , and Proposition 3^ follows im- 
mediately from Corollary 3.7 for all groups except for G = Spn{C). ■ 

Suppose now that G is Spn{C). Then L = GL„(C) and n is the space of 
nx n complex symmetric matrices. We write V for the finite-dimensional space 
of holomorphic polynomials on n spanned by all the minors of the symmetric 

matrix x, and let e be the unitary character of L given by e{l) 



\detl\ 



Lemma 3.8 V is a Spn{C) -invariant subspace of E{l,e). 



Proof. The character e'^ of L is simply |det/|. Therefore, the space I{l,s) 
consists of smooth functions on G = 5p„(C) satisfying 

filng) = det{l)f{g). 

An easy calculation shows that the representation of G on /(l,e) can be 
expressed in the noncompact picture E(l,e) as follows: 



g ■ f{x) — det(a -I- xc) f (^[a + xc] ^ [b + xd]j for g 



G G. 



For p 



G P and w 



-1 

1 



we have 



p-f{x) = det{a ^)f{ab + axa*) 
w ■ fix) = det{x) f i-x-') 

Evidently, transformations of the form x 1 — > c + axa* take minors of x to 
linear combinations of (possibly smaller) minors; thus V is P-invariant. 

Also each minor of x^^ is equal to ±det(x)~^ times the complementary 
minor of x; thus V is w-invariant. Since P is a maximal parabolic subgroup, w 
and P generate G, and hence the space V is G-invariant. 

It remains only to prove that V C E{l,e). Using the Gelfand-Naimark 
decomposition, the functions in V can be lifted to P-equivariant functions on 
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the dense open set PN in G. From the G-invariance of the finite-dimensional 
space V, it foUows that these functions transform finitely under right translations 
by K. Therefore they extend to smooth functions on K, and hence on G. The 
lemma follows. ■ 

Corollary 3.9 ForG = Spn{C), the function det(a;) belongs to the space E{1, e). 
■ 

We can now finish the proof of Lemma BA. 



Proof, (of Lemma 3.4 for G = Spn{C)) For G = Sp„ {€-), we have d = 1, 



and the function ^ is given explicitly by 

^-(0;) =det(l + xx)""/2 



and we have 



Thus 



(x) = det(a;) det{x) 



= det(a:,) det(a5r) det(l + (4) 

To calculate this, we proceed as follows. First of all, it is well known that 
for u a complex symmetric matrix 

det(9„) det(M)'' = const • det(-u)''~^ 

where the constant can be calculated using, for example, the Capelli identity 



from |KS] 



Making a simple change of variables, we deduce 

det(a„) det(l + uY = const • det(l + uY'^. 

Now if w is a fixed n x n complex matrix, then changing variables from u to 
t>*uw, we get 

det(a„) det(l + v^uvY = const • det{vv*) det(l + v^uvY^^- 
This can be rewritten as 

det(5„) det(l + vv^uY = const • det(w'i;*) det(l + vv^uY'^- 
By analytic continuation, we get for all complex symmetric w 

det(9„) det(l + wuY = const • det('u;) det(l + wuY~^- 
Applying this to (^), we obtain 

= const • det{d^) det(x) det(l + xx)""/^"^ 
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The function det(l+a;;E)~"'/^~"'^ is the spherical vector in E{—n~2). Also, by 
the corollary above, det(a;) belongs to E{l,e). Each of these functions extends 
to a smooth function on G with appropriate P-equivariance. By considering the 
equivariance of the product, we deduce 

det(x) det(l + e E{-n - l,e). 

Since D™~^ dei{dx) is a constant coefficient differential operator, we get 

[const Met (a^)] [det(a;) det(l + a;^)""/^"^] <EE{~n-l,e). 
Now in the present case we have d — 1, e = 1, thus we get 
- [d (n - 1) + (e + 1) /2] = > -n - 1 



and so the result follows from Corollary ^J. ■ 
3.2 Proof of the estimate 

We fix k and denote the spherical vector ^k,n — ^-dk by simply $ as before. 
In order to prove the necessary estimates for we first relate it to the "rank 
1" spherical vector 



d- 



We now describe the key result in | DS2 , Theorem 0.1] concerning the function T. 
Let r = '^~2~^ s-iid Kt be the corresponding one-variable if-Bessel function; 
define an Af-invariant function v on the rank 1 orbit Oi = L • yi C n ~ n* by 
the formula 

v{z[m- yi]) = illM for z G R+, me M. 
Then writing d/ii for the equivariant measure on Oi, we have 

vdfii = TdX 

where dX is the Lebesgue measure on n, and^ denotes the Fourier transform 
of tempered distributions. This result is proved in Propositions 2.1 and 2.2 of 
[DS2|. For our present purposes, it is crucial that r depends only on d and e 
but does not depend on n. 

An immediate consequence of 3.5 is the relation 

$ = T^ (5) 

This in turn implies a relation between the Fourier transforms of $ and T which 
we now explain. We start with the following abstract situation: 

Suppose ^ is a Lie group, x is a positive character of A, and B D C are 
subgroups such that each of the homogeneous spaces A/B and A/C admit x- 
equivariant measures drriji/B and drnj^/^j. 
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Lemma 3.10 In the above situation, the space Z = B/C admits a B -invariant 
measure dz . Moreover, the formula 



Cf{aB) = / f{az)dz 



gives a well-defined operator C — Ca.b.c ■ i^/^) ^ i^/B) satisfying 



[Cf] drriA/B 



fdruA/c 



lA/B JA/C 

Proof. By Lemma |5.2| for 6 e -B, c e C, we get 

|detn(Ad5)| , ,_i IdetafAdc) 

: X (C) = 



(6) 



Idett, (Ad 6) I 
Specializing to 6 = c, this implies 

|detb (Adc 



|detc(Adc)| 



|detc (Adc) 



= 1 



and another application of |5.2| proves the existence of an invariant measure on 
B/C. 

The left side of formula (||) gives a x-equivariant mean on the space Cc {A/C) 
thus it agrees with the right side after suitable normalization of the various 
measures involved. On the other hand we have 



\Cf{a)\ 



f (az) dz 



< / \f{az)\dz = C\f\{a) 



therefore for / in Cc {A/C) we get 

/ \Cf\dmA/B< / C\f\dmA/B= / l/M"i^/c- 
Ja/b Ja/b Ja/c 

This shows that C extends to a bounded linear operator from £^ (A/C) 
C} (A/B) such that the formula (^ continues to hold. ■ 
We apply the previous result to the situation where 

A = L,B = S = stab y\C = S' = stab y' 

with = 2/1 + y2 + • • ■ + 2/fc as before, and y' = (yi, y2, • ■ • , yt) G d x . . . x d. 
The space O ~ L/S is the rank k orbit and hence by Lemma 2.1 carries a e^'^^'^- 
equivariant measure. On the other hand, the space Oi x . . . x Oi also carries a 
e^'"''^-equivariant measure, viz. dfj! = d/ii x . . . x d^i; moreover in this situation 
O' ^ L/S' is an open subset whose complement has measure 0. Thus C also 
admits an e^'^'^^'-equivariant measure. Thus by the previous lemma, obtain a 
well defined operator C = Cl,s,S' '■ (C) £} (O) satisfying formula (^. 
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Now given a function / on Oi , we define a function / on O by the following 
two-step procedure: first define / on O' by 

and then set 

/ = cj. 

Then we have the following result: 
Lemma 3.11 For v as above, put g = v — Cv, then we have 

Proof. It suffices to prove 

e-'^--y^g{y)df,{y)^<^>{x). 
To show this, we rewrite the left side as 

J e-<-^'-y')Cv{l-y')d,,{l-y'). 

Now we have 

{x,l- y^) ^ {x,l-yi) -\ \- {x,l- yk) ■ 

Thus setting 

rjil-yi) =exp{-i{x,l-yi)), 

formula (j^) becomes 

Crfudfj.. 



Now using the previous lemma, we can rewrite this as 

k 



(7) 



/ Tjvdfi' = Y[ .f{i-y])v{i- yj) dti-i [i 
J u 



Proof, (of Proposition 3J) In view of the previous lemma, it remains only 
to prove that g e C^{0, d^). In turn, using Lemma 3.10| , it suffices to show that 
V e £^(C, dfi'), or equivalently that 
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This is essentially contained in Proposition 2.1 of [DS2|. The key point is 
that by Proposition 2.3 for /c = 1, we get 



Since the function Kt-(z) has exponential decay at infinity, it suffices to prove 
that the integral on the right converges at 0. For this we note that .^^l^i has a 
pole of order 2t = d — e — 1 at if r > 0, and a logarithmic singularity if t = 0. 
At any rate (dn — 1) — 2t — d {n — 1) + e is greater than —1, which guarantees 
the convergence of the integral. ■ 



3.3 Proof of the estimate 



The key to the proof of Proposition 3.2 is a "stability" result for the function g 



defined in Lemma 3.11. To state this, we temporarily write gk^n and diik,n for 
g and dfi, in order to emphasize dependence on k (the rank of the orbit) and n 



(the rank of the Jordan algebra). Thus Lemma 3.11 becomes 

9k,ndfik,n = ^k,ndX. 



We now recall the notation ni, no, Li, Mi etc., introduced in subsection 2.1 
Thus ni is a Jordan algebra of rank k (with same values of d and e as n). By 
applying the considerations of the previous sections to niwe obtain a family 
of functions gj^k] j — l,...,k, defined on the various Li-orbits in ni. We are 
particularly interested in the function 

9 = dk.k 

which is defined on the open orbit O in ni . Now by definition we have ni C n, 
and moreover we have O d O, where O is the rank k orbit in n. Thus we can 
restrict the function g = gk,n from O to O. The crucial "stability" result is the 
following: 

Lemma 3.12 With the above notation, we have g\^ — I)- 

Proof. The function g is also defined by the analogous two-step procedure 
applied to the Jordan algebra ni. We start with the Mi-invariant function v 
on the rank 1 orbit Oi C fii corresponding to the Bessel function Kr/z'^ . As 
observed after the definition f, the parameter r = (d — e — 1) /2 is independent 
of n. Thus we get 

w|di = (8) 

which is the rank 1 version of the present lemma. 

Next, we consider the open Li-orbit C in Oi x • • • x Ci, and define the 
analogous function v by the formula 

v(l-y') ^v(l-yi)---v{l-yk) 
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for y' = (yi, • • • , y^) € C and I in ii. Comparing this with the definition of v, 
and using formula (||) we deduce 

v\o, = V 

Now the functions g and 'g are defined by the integrals 

g[l.y^) I v{l- z) dz for / in L (9) 
J z 

g • y^) = / V {I ■'z) dz for / in Li 
Jz 

where dz and are the invariant measures on the homogeneous spaces Z = 
S/S' C L/S' = O' and Z ^ {S CMi) / {S' n Li) C Li/ (S" n Li) = C. How- 
ever, as in formula (H) we see that 

S ^ {{S nL^) X Lq) -U, 
S' = ((S"nLi) X Lo) -17. 

Thus in the imbedding O' C O' , we have 

Z = Z. 

Moreover, since both measures are Li-invariant, we have 

dz = dz. 

Thus the two integrals in formula coincide for Z in Li, and the result follows. 
■ 

Let f ^ f denote the inverse Fourier transform which maps functions on ni 
to functions on ni . Thus 



f{y) 



i{x,y) . 



e^^-"' f (x) dX 



where dX is the Lebesgue measure on ni. 

Lemma 3.13 Writing (j) for the Jordan norm polynomial on ni, we have 

. d-(e+l) 
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ld-{e+l) 



Proof. The Fourier transform of tempered distributions is defined by ad- 
jointness from its action on Schwartz functions, and we have the relation 

fdX = fdX. 
Now by the definition of g we have 

gdn = ^k^dX, 
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where dfj. is the equivariant measure on the open orbit O C rii . Propositions ^.2| 
and 2.3 imply that in polar coordinates, the measures dX and dfi are given by 



V/^d^z and Pj^V^d^ z respectively. Thus, writing for the Jordan norm 



polynomial on ni, we get 



d/i — 



d-(e+l) 



dX. 



Combining these formulas we obtain the result. ■ 

Lemma 3.14 Let D be the Cayley operator on rii then for I > we have 



ri+d-{e+l} 



Proof. If f{x) is a Schwartz function on ni and q{y) is a homogeneous 
polynomial on ni, then we have (up to a scalar multiple) 

(dj) = qf, 

where dq is the constant coefficient differential operator on rii with "symbol" q. 
Thus the proof of the Lemma consists in establishing that the above identity 
continues to hold when / (like ^k.k) is a smooth function of polynomial growth 
such that f £ (ni, dX). This is fairly standard; indeed by adjointness we have 
the result 

ddx = dq{!dx), 

where the derivative on the right is the distributional derivative. Under the 
assumption on / , the right side equals {dqf) dX, and the result follows. 



We are now in a position to prove Proposition 3.2 



Proof, (of Proposition 3.2) The function g is M-invariant, thus by Propo- 
sition 2.3, it suffices to prove the convergence of the integral 



\gizm + ■■■ + ZkVkW [Pr^^'Vk] d^z 



By the previous Lemma, this can be rewritten as 



(zij/i + . . . + zkyuf [Pk-'^+'yk] ' dl z. 



Using Proposition 2.3 we can further rewrite this as 

dX where t = d{n — + 1) — (e + 1) . 



e/:^ (ni,(iA) 



Thus it suffices to prove that 



(10) 
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Now the map f ^ f extends as an isometry from £^ (ui , dX) to C"^ (rii , dX) 
(after suitable normalizations of the Lebesgue measures). Thus we have 



/i, /2 e C (m, dA) ^ /1/2 e {n,,dX) ; 



(11) 



we shall deduce (^^ from (|T^) by a suitable choice of /i, /2- 
Let us put 



s = t + 2{e + l~d) = d{n~k~l) + {e 
since n > k, we have s > 0. Now if we set 

/i = /2 = where Zi + I 

then by the previous lemma we have 

I/1/2I =5' 



1); 



h+l2-2{e+l~d) 



(12) 



(13) 



We now consider two cases: if G 7^ S'p„(C) we set li ~ Q and ^2 = s; if G = 
^^^(C), we set li — \ and I2 — s — \. In the former case we have ^1,^2 > 0; 
while in the latter case, we have e = 1, whence s > 2 and ?i, I 2 > 1. Thus in 
either case by the open orbit estimates of Lemmas 3^ and iA , applied to the 
Jordan algebra ni, we deduce that the functions /i and /2 from formula (^2[) 
belong to C'^{ni,dix). Thus formula ([lO|) follows from (13) and (|ll|). ■ 



4 Proof of the main results 



We now explain how to deduce Theorems 0.1 and p.2| from the previous results. 



As explained in the introduction, the arguments are very similar to those in 
|DS1] and [DS2|. Thus we shall limit ourselves to only sketching the proofs 
of the various results below. 



4.1 Proof of Theorem OTT 



In order to prove Theorem 0.1, we introduce a number of spaces. 



First of all, let I {—dk) C C°°(n) be the space of smooth vectors in the 
degenerate principal series defined in section |^.The representation tt — iT-dku 
of the group G on this space is by "fractional linear transformations" , and we 
have 



TT (1) f] (x) = e-'^'"' (0 / (Ad r 1 [x]) for I in L, 



[tt (expx') /] (x) = / (x + x') for x' 



m n. 



By p3| , the space E{—dk) has an irreducible unitarizable spherical {q, K)- 
submodule V which we also regard as a subspace of C°°{n). Thus by Harish- 
Chandra theory, the Hilbert space closure H oi V with respect to the (g, K)- 
invariant norm carries an irreducible unitary representation of G. 
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For convenience, we first describe Ti, as the closure of a G-invariant space. 
For this we introduce the space V consisting of those vectors in I{—dk) whose 
restriction to K, and subsequent expansion in if-isotypic components only in- 
volves the K-types of V. Since V is (g, if )-invariant, the space V is G-invariant 
and we have the following result. 

Lemma 4.1 The functions in V have finite Ti.-norm and Ti. is the closure of 
V. 



Proof. This is a consequence of a general result due to Casselman-Wallach 
on the smooth vectors of a representation. In the present situation, one can also 
give an alternative proof along the lines of the remark in section 2.4 of [ DS2 | as 
follows. 

First of all, the if- types of V have multiplicity 1, and have highest weights 
of the form 



mi7i H h TOfc7fc , 

where mi > ■ ■ • > > and 71, • ■ • ,jk are as in subsection 1.2. Moreover 



the 7i-norm on each if-type is computed explicitly in [ p3| and the ratio of the 
7i-norm to the £^(ii')-norm grows at most polynomially in (mi, • • • ,mfc). On 
the other hand by the Riemann-Lebesgue lemma for / in V, the (isr)-norms 
of its if -isotypic components decay rapidly. Thus such an / will have finite 
7i-norm. Evidently since F C V, the closure of V is as well. ■ 

Next recall the space Ho = ^^{O, dji); by Mackey theory, this space carries 
a natural irreducible unitary representation ttq of P, which is given by the 
following explicit formulas: 

[to (0 V-] {y) = e'^'^^ (0 ^ (Adri [y]) for I in L 
[tto (expx) ?A] {y) — e~^^^'^^ip (y) for x in n , 



where {x, y) is the normalized Killing form of subsection L4. We shall prove 



Theorem D.l by constructing a unitary P-isomorphism T between (7r|p,7i) and 

We first define X on a suitable subspacc of Ti.. For this, let C (G) be the 
convolution algebra of smooth functions on G. Then by standard arguments, 
71" extends to a representation of C (G) on V and we define 

W = 7r(C(G))$C V , 

where $ = $_dfc is the spherical vector in I{—dk). Since G = PK and <I> is 
if-fixed, we also have 

W = TT (C (F)) 
and we shall prove the following result: 
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Lemma 4.2 For each f in W there is a unique X (/) G Tio such that we have 
the equality 

fdx = I{f)d^I, 

of tempered distributions. Furthermore, for all F G C (P) we have 

Iott{F)^ito{F)oI . (14) 

Proof. The key step is, of course, |3.2| which shows that for the function 
$ = ^-dk we have 

$dA = il^dn. 

where il) € Ti-o] or, equivalently, 

^{x) = / e-<''■'yH{y)d^Ji{y). 

JO 



Now for / in L, by Lemma 2.1 we have 

e-*<-'^> [ttoW^] {y)dii{y) ^ e"^'' {I) [ e-^<^'^V(AdrMy])d/i(y) 

Jo 

^ e'^''" {I) e-^'^'"' [l] f e-*<^'^'^'[^l>V(y)dAi(y) 
Jo 

= £-'^'=''(0 / e-<^'''''^^^'y)i;{y)d^,{y)^7^{l)f 



o 



Similarly for x' in n, we have 

'^-■^y^[^To{expx')iJ]iy)d^iiy)= I e-<-+-' i;{y)dfiiy) = n (e^p x') f. 



e 
o 



Thus for any F E C (P), we have 

[TiiF) $] dX^ilno (F) ^]d^i) 
and we can define I by the formula 

X(7r(P)$) =7ro (P)V'- 
Then X satisfies the conditions of the lemma. The uniqueness is clear. 



We can now finish the proof of Theorem 0.1 



Proof, (of Theorem 0.1) Given the previous lemma, the proof of the result 
proceeds along lines similar to pT| and |DS2, ]. By the previous lemma, the 
space Wi = T (W) is a C (P)-invariant subspacc of Ho, and moreover we can 
equip it with a second P- invariant norm, namely that transferred from Ti.. Now 
as explained in 3.3], it follows from Q that Wicontains a further C (P)- 
invariant subspace W2 on which the two norms coincide (up to a scalar multiple 
which we normalize to be 1 by rescaling X) . 
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Since Ho is irreducible, W2 is dense in Ho and thus Ho can be regarded 
as the closure of W2 with respect to the 7i-norm. It follows that the two norms 
agree on Wi as well, and thus the map 

I-^ : Wi ^ W 

extend to an isometric P-invariant imbedding of Ho into H. Now the image 
of J is closed, and contains a G-invariant subspace (namely W); thus since H is 
an irreducible representation, it follows that S is surjective as well. Thus is & 
unitary intertwining operator between {no, Ho) and (7r|p, H). The required ex- 
tension of {tto,Ho) to G is now given by simply transferring the representation 
from (n^H) via J^^. ■ 



4.2 Proof of Theorem ^ 

We now study tensor products of our representations no ■ The analogous study 
for conformal groups of Euclidean Jordan algebras was conducted in [DS1|. 
Since the statements and proofs from |DS1] can be transferred to our present 
(non-Euclidean) setting without substantial changes, we will only sketch some 
of the arguments below. 

Fix s > 2 and a collection of positive integers ki, . . . , fc^ satisfying the con- 
dition 

k = ki + ■ ■ ■ + kg < n. 

For each z = 1, . . . , s, let O* be the L-orbit on n of rank ki, with L-equivariant 
measure dfi^. Let nQi be the unitary representation of G on the space £^(0*, dfi^) 
as described in theorem |0.l| . We wish to study the tensor product representation 

n = noi (8) ... TTcia 

which can be realized explicitly on th e spa ce £^{0^ x . . . x O^, dfi^ x . . . x d/x'*). 
Let Hi, - ■ ■ ,yn be as in subsection 1.3, and define 

Vi = ynii + l + yrn,+2 + . . . + Vrn^+k, , whcrC = fci + . . . + fcj_i , 1 < 1 < S. 

Then Vi is an orbit representative for ; v — vi + ■ ■ ■ + Vs is an orbit represen- 
tative for the rank k orbit O; and the i-orbit of 

v' = {vi,-- ■ ,Vs) 

is an open subset of x . . . x with full measure. We denote by S' and S 
the isotropy subgroups of v' and v, respectively. In the notation of subsection 
2.1, we have v = y^, and thus 



S ={HiX Lo) ■ U. 
It is easy to see that S' can then be written as 

S' = {H[ X Lo) ■ U, 
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where H[ is a certain reductive subgroup of Hi . We now change the notation 
shghtly and write G' for Hi and H' for H'l . 

Example. Take G — -^7(7), s = 2 and ki = 1, k2 — 2. Then fc = n = 3 
and S = G' (the stabilizer of the identity element of n - the exceptional Jordan 
algebra of dimension 27). In this case we have G' — ^4(4) and S' — H' = Spin^ 5 
(cf. p. 119]). 

In general, X — G' /H' is a reductive homogeneous space, and we write 
Ind^/ 1 for the quasiregular representation of G' on £^(X). We decompose this 
using^ the Plancherel measure dp and the corresponding multiplicity function 
m : H ^ {0, 1,2,... , 00}, i.e.. 



/■e 

Ind^/ 1 ~ / m{a)<T dp(a) . 
Jg' 



We define a map Q from irreducible unitary representations of G' to unitary 
representations of P defined as follows 

where Ea denotes the trivial extension of a to S* = (G" x Lq) ■ U, and Xv is the 
unitary character of N defined by 

An easy application of Mackey theory shows that all representations Q{a) are 
unitary irreducible representations of P, and 0(a) ~ 0((t') if and only if cr ~ a' . 

Proposition 4.3 The restriction of II to P decomposes as follows 

n|p~ r m{a)eia)dp{a), (15) 
Jg' 



Proof. This is proved as in [DSl, Lemma 2.1] — here is a sketch of the 



argument. We define an operator F from the space of 11 to functions on P by 
the formula 

[Ff] (In) = Xv {Inl-') /(/ • v'), I e L, n e N. 
It is an easy exercise to verify that F gives a unitary isomorphism 

n|p ~ Indf,^(l(g)x«)- 
Next, using induction by stages we obtain an isomorphism 

Indf,jv(l Xv) ^ Indf^ ((Ind|, 1) ® Xv) ■ 
A final easy calculation shows that 

Ind|, 1 ~ E (^Indg, 1 j ~ y_ TO(cr) (Ea) dp{a) 
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Combining the various isomorphisms, we obtain the result. ■ 

Let K be a unitary representation of G on a Hilbert space Ji, and be a 
subgroup of G. We shall write A{k,, R) for the von Neumann algebra generated 
by the operators {K,{g)\g g R}. If G is a type I group Q, then for irreducible 
K one has A{k, G) = B (H) — the full algebra of bounded operators on H. To 



extend the P-decomposition of 11 from formula (15) to the G- decomposition, 
we require the following 

Proposition 4.4 A(n,G) =A{n,P). 

Proposition iA was proved in |DSl|, 4 .4] for conformal groups of Eucli dean 



Jordan algebras. The proof given in [DS1| combines the low rank theory of |Lil|, 



[Li2| for classical groups, and Jordan algebra techniques for the exceptional 
groups. The arguments extend to our present setting without any significant 
modifications. For the readers convenience, we outline the steps of the argument 



in Appendix 5.2 



Proof, (of Theorem 0.2) Consider the direct integral decomposition of 11 

11= m{K)Kdrj{K) 
into irreducible representations of G. Then 

A{II,P)<Z m{K)A{K,P)dri{K) <Z m{K)A{K,G)dT]{K) = A{n,G). 



The equality of Proposition 4.4 is possible only when the following conditions 



are satisfied (for almost every n with respect to drj): 
• k\p is irreducible (then A{k,P) = A{k,G)); 



If K,\p ~ k'\p, then k ~ k' . 



In other words, in this case (almost) every irreducible representation Q{a) 
from the spectrum of n|p extends uniquely to a certain irreducible representa- 
tion of G, which we denote by 9{cr); and the P-decomposition ( p^ ) gives rise to 
the G-decomposition 

n= r m{a)0{a) dp{a) 
Jg' 

and the theorem follows. ■ 

Example. Again, take G = -£7(7), s = 2 and ki = 1, ^2 = 2. Then the 
map (7 9 {a) establishes a correspondence between the spectrum of 11 and the 
spectrum of the rank 1 reductive symmetric space P4(4)/5'pm(4, 5). In other 
words, we obtain a duality between (some subsets of) the unitary duals of two 
exceptional groups: split F4 on one side and split on the other side. As 
with Howe's duality correspondence (the usual 6'-correspondence) , we expect 
that this new duality will have smooth and global analogues. 
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5 Appendix 



5.1 Measures and Jacobians 

If : X — > y is a diffeomorphism between manifolds and dx is a measure on 
X, the pushforward of dx is the measure F^^dx on Y defined by 

/ f[F^dx\ = / J{F{x))dx. 
Jy Jx 

If d\ is the Lcbcsguc measure on R" and A : U V isa diffeomorphism between 
open sets in M", then the "change of variables" formula for the Lebesgue integral 
says 

/ f {A {x))\det Da {x)\dX= f fdX 
Ju Jv 

where Da {x) : R" — > K" is the differential of A at a; in U. In other words, we 
have 

A* [|det Da {x)\ dX] = dX or 

or equivalently 

A^^dX = gdX where g (A (x)) = |det Da {x)\~^ 

A measure on an open set i7 C R" will be called regular if it is of the form 
(j)dX where (/> is a smooth positive function. If A : U ^ V is a diffeomorphism, 
then we have 

A^ {(f>dX) = ipdX where ii {A {x)) = (j){x) / |det Da {x) \ ; 

thus ^* maps regular measures to regular measures. If dx is a measure on a 
smooth manifold X and C/ is a coordinate open set, then the pushforward of 
dx\u under the coordinate map is a measure on an open set in R", which we 
shall call the "local expression" of dx. We shall say that dx is regular if each 
of these local expressions is a regular measure in the earlier sense. It follows 
from our discussion that if : X — > y is a diffeomorphism and dx is a regular 
measure on X, then F^dx is a regular measure on Y . 

Suppose X and Y are manifolds with regular measures dx and dy, and F : 
X ^ Y is a. diffeomorphism. The Jacobian of F is the function Jp (x) = 
JF,dx,dy (x) on X satisfying 

J f{y)dy = J JF{x)f{F{x))dx. 

or, equivalently 

-F* [Jpdx] = dy 
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If we have another difFeomorphism G : Y Z, where Z is a manifold with 
regular measure dz, then 

Jgf (x) = Jf{x)Jg (y) where y = F {x) (16) 

If X and Y are open sets in M", and dx — (pdX, dy — ipdX, then it is easy to see 
that 

_ IdetZ^F {x)\^p{x) 

F.dx,dy — 



cj^iFix)) 

If X and Y are smooth manifolds, then we can determine the Jacobian in 
a similar manner by passing to local coordinates and using formula (16). In 
particular we see that for regular measures, the Jacobian is a well-defined smooth 
positive function. 

The following lemma will be quite useful in calculating Jacobians. 

Lemma 5.1 Suppose X and Y are manifolds with regular measures dx and 
dy; let x £ X , y £ Y , and fix linear bases for the tangent spaces T^X , TyY . 

Then there is a positive constant c, such that for any diffeomorphism X —t Y 
satisfying F [x) — y, we have 

Jf{x) = cjdetDi? {x)\ 

where we regard the differential Dp (x) : T^X —f TyY as a matrix for the above 
bases. 

Proof. If X and Y are open sets in R", and dx — (j)dX, dy — ipdX, then 

Idet Dp (x)\ih (x) ^ , M 

JF.dx,dy = i-f^^^^ =c deti^F X , 

where the determinant is computed for the standard basis of T^X — TyY = R". 
If we use different bases then the scalar c is replaced by a different scalar, which 
is still independent of F. Passing to local coordinates, we obtain the result in 
general. ■ 

Now suppose X is a smooth manifold with the action of a Lie group G, and 
let X be a positive multiplicative character of G. A regular measure dx is called 
X-equivariant 

g^,dx — X (g) dx for all g in G. 

Equivalently 

Jg,dxMx (x) = xigy^ for all X e X,g e G. 

For example, the Lebesgue measure dX on R" is equivariant by the character 
Idetf^ of the group GL (R"). 
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If X is a homogeneous space for G then for x in X, the tangent space T^X 
can be naturally identified with g/g^, where g^ is the Lie algebra of the stabilizer 
of X. Moreover, for 5 in G we have 

^g = Adg:g/g-^g/g^-. 

Lemma 5.2 A G -homogeneous space X admits a x-equivariant measure if and 
only if 

X{h)'^ = |dctj,/gx {kdh)\ for all h E G^. 

This result is well-known and can be proved in a manner analogous to The- 
orem 1.1.9 in Q. Here is a quick argument for the necessity of the condition. 
If h is in G^ , then we have 

X = Jh (x) = c |det Dii\ = c |detj|/gx (Ad h) | . 

Specializing to ft, = 1 G G, we deduce that c = 1. 

Corollary 5.3 If G is reductive, then the condition of the previous lemma be- 
comes 

X{h) = Idetjjx (Ad/i)| for all h e G"" . 
5.2 Low rank representations 

Let r be a unitary representation of G. Consider its restriction to P, and its 
further restriction to N . Since N is abelian, the restriction t\m decomposes 
into a direct integral of unitary characters of N . This decomposition defines a 
projection valued measure on the dual space TV*, which we identify with n. If 
this measure is supported on a single non-open orbit Or C N , we say that r a 
low-rank representation of G and write 

rankAT t = r. 

An element xi is a primitive idcmpotent in a Jordan algebra TV, and we can 
consider the associated Peirce decomposition 

iV = N{xi, 1) + Nixi, + N{xi,0). 

Observe that the spaces N{xi, 1) and iV(a;i, 0) are the Jordan algebras of ranks 
1 and n — 1, respectively, with the respect to the Jordan structure inherited 
from N. 

We will write iVi and A^o for N{xi, 1) and N{xi,0), respectively. Similarly, 
we write Gq for the conformal group of Nq, Pq = LqNq for the Siegel parabolic 
subgroup of Go, etc. 

Below are the examples of Nq and Go for several different groups G : 
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• For G = Op+2,p+2, we have Nq = M. (rank 1 Jordan algebra), and Go = 
GL2(K). 

• If G = Spn,n , then Gq = Spn-l.n-l- 

• If G = £^7(7), then iVo = M^'*^ (rank 2 Jordan algebra), and Go = Oe.e- 
. If G = EriC), then Gq = Oi2(C). 

Set f = 0"^2 0^'~^' ® ®"=2 0^'^^" and n' = f + ni. Then n' is a two-step 
nilpotent subalgebra of g with the center rii . 

Any generic unitary irreducible representation of the group N' is determined 
by the unitary character of its center A^i . We denote by pt the unitary irreducible 
representation of N' which restricts to the multiple of the character xt on A^i , 

teN^ = Nl\{0}. 

Consider now a subgroup GqN' of G. We can view Go as a subgroup of a 
symplectic group Sp{f) associated with the standard skew-symmetric bilinear 
form on f. Hence we can use the oscillator representation of Sp{f) to extend the 
representation pt of N' to a representation of GqN' which we denote by pt- 

Let cr be a unitary representation of G, rankjv a = r, < r < n. Without 
loss of generality we may assume that a has no iVi-fixed vectors. Then by 
Mackey theory, we can write down the decomposition 

(^\goN' = Kt <8) ptdt, 

where all Kt are unitary representations of Gq. 



Proceeding as in |DS1, 3.1], we verify that all of the representations Kt are in 



turn the low-rank representations of Gq- More precisely, we have the following 

Lemma 5.4 Let a be a low-rank representation of G, rankjv a = r, < r < n. 
Then for any t G the No-spectrum of the representation Kt is supported on 
a single Lq- orbit, and rank^rg Kt = r — 1. 

The next technical lemma is proved exactly as in | ]DSl| , 3.2]: 

Lemma 5.5 If for all t E one has A{Kt, Gq) = A{Kt, Pa), then 



Ptdt, Go C ^ / Kt® ptdt, PoN' 



Theorem 5.6 Let a be a representation of G, ranker cr — r, Q < r < n. Then 

A{a,G) ^A{<J,P). 
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Proof. The proof of the theorem is based on the fact that Go and P generate 
G, so it is enough to verify that A{a, Go) C A{a, P). Since PoN' is a subgroup 
of P, by the Lemma above the assertion of the theorem is equivalent to the 
claim that for any t £ 

A{Kt,Go) = A{Kt,Po). 



By the Lemma 5.4, all nt have rank r — 1. Proceeding in the same manner, 
we reduce the statement of the theorem to that about rank representations of 
the certain (classical) group Goo- Since all rank representations are the direct 
integrals of characters, and any character of Gqq is determined by its restriction 
to the Siegel parabolic Pqq C Gqo, the theorem follows. ■ 
We now consider the tensor product 



n = ttqi 



ior k = ki + k2 + ■ ■ ■ + kg < n. Then 11 is low-rank representation of G and 
rankAT 11 = fc. Applying the theorem above to H, we obtain the statement of 
Proposition 4.4 for k < n. 

It remains to check Proposition 4.4 for k = n. For all groups G except 
Op+2,p+2, Op+4(C) and £^7(7), Et{C) the statement of the proposition follows 



from the results of |Li2], since all the representations form the spectrum of H 
appear in the Howe duality correspondence for appropriate stable range dual 
pairs (G^,G). For the exceptional cases listed above, the argument can be 
constructed along the lines of Section 4 of [DS1|. 



5.3 Tables of groups and symmetric spaces 

In the table below we list the various groups G arising from the Tits-Kantor- 
Koecher construction for non- Euclidean Jordan algebras; together with the as- 
sociated symmetric spaces K/M and L/H; and the crucial root multiplicities d 
and e in S(tc, 4c)- The rank of the Jordan algebra is n in each case, except for 
Op+4(C), Op+2,p+2 where the rank is 2, and £7(7), Et(C) where the rank is 3. 



G 


K/M 


L/H 


d 


e 


Gi2„(M) 


02„/(0„ X On) 


GLn{R)xGLn{R)/GL4R) 


1 





02n,2n 


{02n X 02n)/02n 


GL2nm/spnm 


2 





£7(7) 


SUs/Spi 


M*X^6(6)/£4(4) 


4 





Op+2,p+2 


[Op+2]V[Oi X 0^+1] 


M* X Op+i,p+i/Op,p+i 


P 





Spn{C) 


Spn/Un 


GL„(C)/0„(C) 


1 


1 


Gi2n(C) 


U2n/{Un X [/„) 


GL„(C)xGL„(C)/GL„(C) 


2 


1 


04„(C) 


04n/U2n 


GL2n{C)/SpniC) 


4 


1 


EriC) 


Ej/{Ee X Ui) 


C*xEe{C)/F4{C) 


8 


1 


Op+4{C) 


Op+i/{Op+2 X C/i) 


C* X Op+2(C)/Op+i(C) 


P 


1 


Spn.n 


{Spn X Sp„)/Spn 


GL„(H)/0; 


2 


2 


Gi2n(H) 


Sp2n/{Spn X Spn) 


GL„(H)xGi„(H)/Gi„(H) 


4 


3 
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In this table we list the various homogeneous spaces X = G' /H' that arise in 
the ^-correspondence of Theorem 0.2. 



G 


X 




r^T IW\ 1 \r^T (V\ V CT ('TOM 




5pfc(M)/[5pfc,(M) X ... X SvkAM 




Spin^,^/ Spin4^4 (fci = 1, fc2 = 1) 
Fm)/SpinA^b (fci = 2, fc2 = 1) 


Op+2,p+2 


SOp^p+i/SOp^p (fci = l,fc2 = 1) 


SpniC) 


Ofe(C)/[Ofe,(C)x...xO,^(C)] 


GL2n{C) 


GLk{C)/[GLkAC) X ... X GLkAC)] 


04„(C) 


Spk(C)/[SpkAQ X ... X SpkAQ] 


E7{C) 


SpinglC) / Spins{C) (fci = l,fc2 = 1) 
Fi{C)/Sping{C) (fci = 2, fc2 = 1) 


Op+4(C) 


SOp+i{C)/SOp{C) (fci = l,fc2 = l) 


Spn,n 


01/ Ofc^ X . . . X 01^ 


GL2„(H) 


Gife(H)/ [Gife,(H) X ... X Gife, (H)] 
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